destination-snowflake: get tests to pass - durably #45370
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @stephane-airbyte and the rest of your teammates on |
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
2 times, most recently
from
7b66324 to
8c7ebea
Compare
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
3 times, most recently
from
ea6583e to
bd98bf7
Compare
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
3 times, most recently
from
5d73a2f to
78a1292
Compare
| @@ -317,8 +317,9 @@ constructor( | |||
|
|
|||
| fun main(args: Array<String>) { | |||
| IntegrationRunner.addOrphanedThreadFilter { t: Thread -> | |||
| if (IntegrationRunner.getThreadCreationInfo(t) != null) { | |||
| for (stackTraceElement in IntegrationRunner.getThreadCreationInfo(t)!!.stack) { | |||
| val threadCreationInfo = IntegrationRunner.getThreadCreationInfo(t) | |||
There was a problem hiding this comment.
there was a race condition here, where getThreadCreationInfo would return nun-null on the 1st call but null on the 2nd call.
There was a problem hiding this comment.
😅
do we want to track down why this method sometimes returns null? Or does it go away in the new cdk?
There was a problem hiding this comment.
I know why :)
The reason being that between the time the IntegrationRunner looks at all live thread and the time we hit this code, the thread could have completed, and all its threadLocal could have been garbage collected, because their only durable instance is held in the thread... so what happens is that we called getThreadCreationInfo(t), which was not null. Then the thread completed, the map was cleared, and we call again getThreadCreationInfo(t), but at this point it returns null... The magic of parallel computing
There was a problem hiding this comment.
thought about it a bit more, and I think the proper fix is not to pass in a Thread to the filter, but instead a ThreadInfo, which would contain a Thread, but also all the info we might need (like currentStackTrace and ThreadCreationInfo), so that the thread itself can die after this is getting called without causing any side effects.
| @@ -1 +1 @@ | |||
| JunitMethodExecutionTimeout=30 m | |||
| JunitMethodExecutionTimeout=5 m | |||
There was a problem hiding this comment.
the sum of all the tests runs in about 10 minutes. The reason some were timing out at 30minutes was because of all the extra table in the destination database. Now, we're cleaning those as we run tests, guaranteeing that overall the tests will be fast
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| open class Batch(val name: String) |
There was a problem hiding this comment.
not sure where that's coming from. Deleting
There was a problem hiding this comment.
probably me doodling cdk stuff. thanks for removing
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
2 times, most recently
from
f61c39f to
fcdf428
Compare
stephane-airbyte
changed the title
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
from
fcdf428 to
22ae7e0
Compare
octavia-squidington-iii
added
the
area/documentation
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
from
22ae7e0 to
4e1a3e6
Compare
stephane-airbyte
changed the title
stephane-airbyte
force-pushed
the
stephane/09-13-change_signature_of_orphanedthreadfilter
branch
from
fffb246 to
90e3ffd
Compare
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
from
50ce747 to
894aa5f
Compare
stephane-airbyte
force-pushed
the
stephane/09-13-change_signature_of_orphanedthreadfilter
branch
from
90e3ffd to
f49e6a6
Compare
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
from
894aa5f to
dc20330
Compare
stephane-airbyte
force-pushed
the
stephane/09-13-change_signature_of_orphanedthreadfilter
branch
from
f49e6a6 to
90c1a66
Compare
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
from
dc20330 to
db565bc
Compare
stephane-airbyte
force-pushed
the
stephane/09-13-change_signature_of_orphanedthreadfilter
branch
from
90c1a66 to
c35540c
Compare
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
from
db565bc to
066ddf7
Compare
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
from
066ddf7 to
e01bb55
Compare
stephane-airbyte
changed the base branch from
stephane/09-13-change_signature_of_orphanedthreadfilter
to
graphite-base/45370
stephane-airbyte
force-pushed
the
graphite-base/45370
branch
from
c35540c to
6d74db7
Compare
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
from
e01bb55 to
1e3f4b1
Compare
stephane-airbyte
changed the base branch from
graphite-base/45370
to
master
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
from
1e3f4b1 to
8d93a3a
Compare
stephane-airbyte
force-pushed
the
stephane/09-10-destination-snowflake_bump_cdk
branch
from
8d93a3a to
a4d667f
Compare
Merge activity
|
stephane-airbyte
deleted the
stephane/09-10-destination-snowflake_bump_cdk
branch
TL;DR
Make destination-snowflake pass all tests
What changed?
How to test?
Why make this change?
These changes aim to improve the reliability and maintainability of the Snowflake destination connector. The updated CDK version and reduced test timeout should lead to faster and more efficient testing. The enhanced error handling and cleanup processes will help in identifying issues more quickly and keeping the test environment clean. Removing unused classes reduces code clutter and improves overall code quality.